fix(ai): stated provider flags win, keys for two providers stop, github never inferred - #857

Description

Design note.

• Who. An operator running fluid forge, fluid init, fluid ai test or fluid doctor on a machine that may hold key variables, a saved fluid ai setup choice, keyring keys and a local Ollama. Also CI runs without a TTY. Command Center always names its provider.
• Primary journey. Name a provider (--llm-provider or FLUID_LLM_PROVIDER) and every command uses exactly that provider with its own key. Nothing you state (provider, model, endpoint) is replaced by a setup pick.
• One ladder on every surface: named provider, then exactly one provider's key variables, then the saved choice, then exactly one keyring key, then a running Ollama. Forge, doctor, interactive setup and fluid ai test share it, and there are no new flags or variables.
• States:
  • Nothing configured: unchanged.
  • Ambiguous (keys for 2+ providers): every surface says "No LLM provider is selected: X, Y are all set, so none of them is chosen. Pick one with --llm-provider or FLUID_LLM_PROVIDER." Forge without a TTY errors; doctor shows "Not selected" plus that next step; interactive forge offers guided mode; init exits 1; ai test exits 2.
  • Resolved: the saved choice or the one keyring key decides, and the run says which ("Using Google Gemini, the one provider with a keyring key.").
  • Success: "AI: / ". The welcome and interview panels now name that same provider.
• Messages say what happened and what to do next, and name variables, never values. A named coding agent gets "fluid ai test does not test …" rather than "Unsupported AI provider".
• Docs and help. CLI help is unchanged; the forge_docs pages are listed under Docs TODO.

Four provider-selection fixes, one commit each. After those comes a merge of main (the 0.29.0 changelog cut), then two commits from this PR's defect scan (effbae02, 523cb579), one behaviour-preserving refactor that ripwire --quality-delta asked for (ee5c679c), and a merge of main after #856 (c30f4476, CHANGELOG entries from both sides kept). The founder confirmed the edge cases on 2026-11-12.

1. Keys for two or more providers stop the run (6b65247a).

Before: with OPENAI_API_KEY and ANTHROPIC_API_KEY both set and nothing naming a provider, a run without a TTY skipped both keys and used a running local Ollama. The same happened with one key plus OLLAMA_HOST. Founder decision: stop and name the keys.

Now provider_resolve._infer_provider_from_ambient returns no provider before it probes Ollama when the key variables name two or more providers. Forge stops with copilot_no_llm_provider, for example "No LLM provider is selected: OPENAI_API_KEY, ANTHROPIC_API_KEY are all set, so none of them is chosen. Pick one with --llm-provider or FLUID_LLM_PROVIDER." fluid doctor reports the same. The saved fluid ai setup choice and exactly one keyring key still come first, as before. GEMINI_API_KEY with GOOGLE_API_KEY counts as one provider.

2. Interactive setup and fluid ai test read the ladder's key variables, never github (99b9be7c).

Before: inline setup's step 2 took any PROVIDER_ENV_VARS variable.

• GITHUB_API_KEY alone selected github, which setup then failed to build (KeyError: 'github'). The comment on PROVIDER_ENV_VARS says github is only used when named.
• It ignored GEMINI_API_KEY.
• With two keys set, it used the first.

Now step 2 reads the same key variables as a run without a TTY (_explicit_key_signals):

• one provider's key selects it, with GOOGLE_API_KEY read before GEMINI_API_KEY as the resolver does;
• OLLAMA_HOST is left to step 3;
• with keys for two or more providers, exactly one keyring key decides, as in the ladder; otherwise setup prints the message and stops.

fluid ai test and fluid init's "is AI configured" check follow the same rule. GitHub Models stays reachable with --llm-provider github.

Also in this commit:

• A test-isolation fix: tests/conftest.py::_session_env_is_restored. The import tests' canary key leaked into os.environ through set_session_env, and later tests then selected openai.
• No .secrets.baseline change at HEAD. Commit 2 refreshed one line number, and the refactor moved that line back, so the merge of fix(ai,receipts): ai models --provider takes the LLM registry; run ids never repeat #856 restores main's file.

3. Interactive fluid forge and fluid init keep the provider, model and endpoint you state (636eb62a).

Before: after run_ai_setup_inline, forge overwrote args.llm_provider, args.llm_model and args.llm_endpoint with setup's pick, and so did init's AI path. So fluid forge --llm-provider ollama could call Anthropic. The same went for FLUID_LLM_PROVIDER, FLUID_FORGE_AGENT, FLUID_LLM_MODEL and FLUID_LLM_ENDPOINT.

Now:

• A run that names its provider (providers.named_llm_provider) skips inline setup and resolves as a run without a TTY does. The named provider gets its catalog model unless --llm-model is given (fix(ai): saved AI choice in non-interactive runs, JSON-only --agent stdout, mission agent loop #854's rule: a named provider takes nothing from the saved choice).
• Otherwise providers.fill_llm_args_from_setup fills only what the run leaves unstated, and records the pick as not named, so FLUID_LLM_API_KEY stays scoped (security(ai): FLUID_LLM_API_KEY only for a named provider; saved keys stay off project hosts #853).
• An agent that setup found is dropped when the run states an endpoint or a model, or runs the agent loop.
• The welcome panel and the interview's "What I see" panel name the provider the run uses.
• fluid init's handoff no longer offers AI setup when a provider is named.

4. fluid ai test honours FLUID_LLM_PROVIDER and FLUID_FORGE_AGENT (99933fe6).

Before: it read only --llm-provider, so with FLUID_LLM_PROVIDER=anthropic it tested the saved provider, which forge would not use.

Now it reads the provider the run names, as forge does, and that provider gets FLUID_LLM_API_KEY, as with the flag. A coding agent or mcp-sampling named either way is not tested, and the command says so. Before, claude-code failed as "Unsupported AI provider 'claude_code'".

Fixes from the PR gate (defect scan of this diff):

• effbae02: fluid init's post-init check imported llm.provider_resolve before llm.providers. The two modules import each other, so in a plain fluid init, which loads neither before that check, it raised ImportError, and the "create your first data product?" offer was dropped. Main is fine here only because it did not import that module there. The helpers are now re-exported from llm.providers and imported from there. A test runs each entry point (init, ai test, setup) in a fresh interpreter. The same commit makes fluid ai test's "does not test" guard accept every spelling forge accepts (MCP-Sampling, Mcp_Sampling).
• 523cb579: with key variables for two or more providers, fluid init's AI path used to write the customer-360 template and exit 0, in CI too. It now exits 1 after setup has named the keys, as forge does without a TTY. Please confirm this choice: the alternative is to keep the template fallback, as interactive forge keeps its "Continue with guided mode (no AI)?" offer. With nothing configured, init still offers the template.

Command Center is unaffected. fluid_cli_service.forge_from_intent always passes --llm-provider with that provider's own key variable; checked at CC origin/main 88067918. CC's doctor call does not read the AI-readiness rows. CC's exact drafting argv (--llm-provider anthropic, with OPENAI_API_KEY and ANTHROPIC_API_KEY both set) still uses anthropic, checked live.

Prior art (borrow receipts):

• simonw/llm @ 8b475ffdfc2d: an explicit -m / --key beats LLM_MODEL and the env key, which beat the saved default (docs/setup.md). Fix 3 has the same precedence.
• aider @ 5dc9490bb35f: aider/onboarding.py::try_to_select_default_model takes the first key it finds in a fixed order, which ignores the others silently. Fix 1 diverges from this on purpose, by the founder's decision.
• LangChain init_chat_model raises ValueError rather than guessing a provider it cannot infer. Google's Gemini docs say GOOGLE_API_KEY wins when both Gemini variables are set, which step 2 follows.

Review verdicts (/ship)

• security-review (size L, Opus judge): no findings on the branch. A re-review of the fix delta (99933fe6..HEAD) found none either.
• defect-scan (diff, hunter, then skeptic, then referee): 5 candidates.
  • 2 are pre-existing and listed under follow-ups.
  • 1 was refuted (whitespace-only FLUID_LLM_PROVIDER).
  • 2 were fixed (effbae02, 523cb579).
  • The referee found one more, the circular import, also fixed in effbae02.
  • Re-check: all 3 fixes verified. The refactor gave identical output across 15 selection scenarios against the pre-refactor tree. No new defect.
• world-class-inspection (diff, delivery review): PASS. The delivery review (ln-52) traced all 4 acceptance items to their implementation and to observed behaviour, with no P0-P2 finding. The pre-setup preflight it removed is still covered (forge_modes.run_ai_copilot_mode preflights before any keyring read). The old PROVIDER_ENV_VARS loops are gone, with no dual path left. Overall verdict: PASS.
• UX (changed CLI journeys): 9/10, no severity-3+ issue. Two P3s are follow-ups:
  • init's remedy names --llm-provider, which fluid init lacks (FLUID_LLM_PROVIDER works);
  • fluid ai test adds its generic "Run fluid ai setup" hint even for a named agent.
• ripwire --quality-delta (fresh origin/main tree plus this diff):

  • First run: 36 gating rows. The refactor in ee5c679c removed the complexity and length growth in run_ai_setup_inline, _resolve_ai_test_config and init._ai_mode, and the test duplication.

  • Still gating:

    • test-scaffolding token clones of 18-64 tokens (LlmConfig builders, env-cleaning loops, Namespace helpers);
    • the _stated_arg idiom;
    • short-horizon churn on functions that fix(ai): real endpoints, --agent AI mode, claude auth errors, keyless auto-pick, an isolated claude -p #850-fix(ai): saved AI choice in non-interactive runs, JSON-only --agent stdout, mission agent loop #854 and this PR all touched.

    None is a production-code regression. Changing them would only game the number.

• Rounds: 1 fix round, plus the quality-delta refactor.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would break existing functionality)
  • With keys for two or more providers and nothing naming one, forge no longer falls back to a running Ollama, and interactive setup no longer uses the first key. Name the provider with --llm-provider <name> or FLUID_LLM_PROVIDER=<name>.
  • OLLAMA_HOST together with a cloud key now counts as two providers.
  • Interactive fluid forge --llm-provider X now uses X with its catalog model, not the saved provider's model.
• Documentation update
• Refactor / code cleanup
• New provider or provider enhancement

Documentation

• Docs PR linked:
  • Docs PR:
• No docs needed — this change does not affect user-facing behaviour
  • Justification:
• Docs TODO — I know docs are needed and will create a PR in forge_docs before this is merged

forge_docs pages to update:

• docs/advanced/llm-providers.md
  • "Which provider a run uses" and "When nothing is configured": key variables for two or more providers stop the run instead of reaching a running Ollama, unless the saved choice or exactly one keyring key decides. OLLAMA_HOST counts as a key variable. The line-113 sentence "an Ollama that answers ... is still used" changes.
  • A named provider skips interactive setup, and a stated model or endpoint wins.
  • Setup reads GEMINI_API_KEY, and github is used only when named.
• docs/cli/forge.md: interactive runs keep --llm-provider / --llm-model / --llm-endpoint and their FLUID_LLM_* twins.
• docs/cli/ai.md:
  • fluid ai test honours FLUID_LLM_PROVIDER / FLUID_FORGE_AGENT, which get FLUID_LLM_API_KEY;
  • it does not test coding agents or mcp-sampling;
  • keys for two providers stop it.
• docs/cli/doctor.md: readiness says "Not selected" and names the keys when two providers' keys are set.
• docs/advanced/environment-variables.md: FLUID_LLM_PROVIDER / FLUID_FORGE_AGENT apply to fluid ai test too; OLLAMA_HOST takes part in the two-provider rule.
• docs/advanced/production-troubleshooting.md: copilot_no_llm_provider with "... are all set" now appears even while Ollama runs.
• docs/upgrading.md: bullets under "On forge-cli main since 0.29.0 (not yet released)" for the three breaking points above.

Checklist

• I have read the Contributing Guide
• My code follows the project's coding standards
• I have added tests that prove my fix/feature works
• All new and existing tests pass (pytest)
• I have run ruff check and black with no errors
• New maintained Python files include license headers (python scripts/add_license_headers.py; examples/** exempt)

Testing

• New tests/test_llm_provider_selection_flags.py: 43 test functions, 50 cases. On origin/main, 37 cases fail and 13 regression guards pass. Each of the four fix commits was replayed on a clean origin/main checkout and is green on its own: 616, 627, 642, then 648 related tests.

• Full suite on the four fix commits: 34,884 passed. A rerun on this head is in progress, and CI runs the suite too. The 13 failures are the shared venv's known baseline:

  • dlt dotenv: 3 + 1;
  • the duckdb sandbox under macOS /var: 2;
  • test_lazy_core_commands: 2, because the probe imports the venv's editable install;
  • test_startup_budget: 5.

  One more, test_iac_databricks_metadata_fake_e2e, failed on a broken pipe to its local fake server under load and passes alone. Before the two gate-fix commits, the 112 related test files passed (2716). After them, the three most affected files passed: 137.

• Startup budget in a lean .[dev,local] venv installed from this branch: 9 passed (build_parser() 299/314, discover_providers() 188/198, fluid providers 311/327).

• Gates:

  • ruff, the dead-code sweep, black 24.10.0 and product_guardrail: green. The F401/F841 hits on the changed files are unchanged from base.
  • License headers and tests/test_license_headers.py: pass. lint-imports: 6 contracts kept.
  • detect-secrets 1.5.0 on the changed files: rc 0 locally, and rc 0 in python:3.12-slim (CI's recipe) with the baseline left unchanged.

• Reviews:

  • An independent diff verifier: 5 findings, all fixed in the four commits.
  • A defect scan of the diff (hunter, then skeptic, then referee):
    • 2 pre-existing and listed below;
    • 1 refuted (a whitespace-only FLUID_LLM_PROVIDER);
    • 2 fixed in effbae02 / 523cb579;
    • plus the circular import above, which the referee reproduced in a fresh interpreter.
  • A security review (Opus): no findings. The new "named" signal reads only protected selectors (--llm-provider, FLUID_LLM_PROVIDER, FLUID_FORGE_AGENT), so a project .env cannot unlock FLUID_LLM_API_KEY. Setup's pick is still recorded as not named. The ambiguity branches send no operator key to a project-controlled host.

• Live A/B through the fluid CLI against main:

  • Setup: env -i, a temp HOME, the null keyring, HTTP(S)_PROXY at a dead port (no egress), a PATH without agent CLIs, and a loopback stub that logs the path, model and key tail of each request.
  • Fix 1: fluid doctor with both keys and the real Ollama visible: main reports ollama ready, this branch "Not selected" plus the message. forge --non-interactive with both keys and OLLAMA_HOST at the stub: main sent 2 generations to Ollama; this branch stops naming all three variables, and the stub sees 0 requests.
  • Fix 2: GITHUB_API_KEY only. Interactive forge on main prints "Using github from environment." then KeyError: 'github'; this branch infers nothing. fluid ai test: main "Unsupported AI provider 'github'", this branch "No AI provider configured".
  • Fix 3: interactive fluid forge --llm-provider ollama --llm-model stated-model-x --llm-endpoint <stub>, with a saved Anthropic and its key. Main printed "Using Anthropic (Claude) from environment." and called Anthropic; this branch sent its 2 requests to the stub with stated-model-x. The same holds with FLUID_LLM_PROVIDER / FLUID_LLM_MODEL / FLUID_LLM_ENDPOINT. --llm-model with a saved Ollama: main sent the saved model, this branch the stated one.
  • Fix 4: FLUID_LLM_PROVIDER=anthropic with a saved openai: main tested openai, this branch anthropic. FLUID_FORGE_AGENT=claude-code gets the "does not test" message.

Follow-ups found, all pre-existing (same on main) and not fixed here:

• --llm-endpoint / FLUID_LLM_ENDPOINT never reach litellm for hosted providers. litellm_backend._apply_ollama_api_base sets api_base for Ollama only, so --llm-provider openai --llm-endpoint <host> still calls api.openai.com. This was checked live.
• A copilot stop under --non-interactive prints a full traceback before the error line.
• fluid ai test uses the saved model for a named provider. It also tries the saved choice before a single key variable (the ladder does the reverse), and reads GEMINI_API_KEY before GOOGLE_API_KEY (forge reads GOOGLE_API_KEY first).
• fluid doctor reports "Unknown LLM provider" for claude_code, Claude-Code and mcp-sampling, although forge accepts them.
• init._offer_first_forge copies args.llm_provider into a new forge Namespace without the llm_provider_named record. If that handoff ever ran after _ai_mode filled in setup's pick, the pick would count as named for FLUID_LLM_API_KEY. Today it does not run there: a written contract skips it, and a declined preview returns 1.
• Setup prints "Using Ollama (saved-model)" before a stated --llm-model wins. The next line, "AI: …", is correct.